Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient ''labeled'' pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on unlabeled data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training data with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data.

We thank the reviewers for their comments and endorsements. Below are our answers to the main questions/concerns. R1: Training on test-fMRI samples - not convinced the approach is valid. We understand the reviewer's concern. Note however that our "training on test data" refers only to training on We will better clarify the distinction between training on the "test-fMRI" (which is the input to the network, We realize that this distinction is confusing, and will clarify it.

The paper's writing and figures are of very high clarity and quality. The method is novel and the basic innovation is in the new objective function, which has encoder-decoder dynamics that are intriguing. The area of research is tackling the difficult problem of trying to reconstruct images from human brain activity with recent machine learning and neural network techniques, which is a strong fit for the NeurIPS conference. The results in Figure 4e) are impressive and look like a convincing improvement over Shen et al. 2019 as they do not need a generative model prior at all, but train an end-to-end architecture. The only ImageNet statistics in their network are pretrained low-level AlexNet features (thus also further lowering the potential influence of category set statistics).

Dear authors, congrats on the acceptance-- this paper was discussed extensively, the the reviewers provided multiple comments and feedback-- please do take the feedback and requests of all the reviewers into account when preparing your final manuscript. In particular, it would be important to clearly describe in what settings (i.e.

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient ''labeled'' pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training data with both types of unlabeled data.

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient ''labeled'' pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training data with both types of unlabeled data.